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1. INTRODUCTION 

Increasingly advanced technology has led to an enormous variety of mobile-based services, 
particularly mobile-based services that using smartphones. 2 types of multi-user mobile-based services with 
lots of users are personal cloud computing and instant messaging. Cloud computing is a technology services 
that are offered by the cloud service provider (CSP), among other types of deals platform as a service (PaaS), 
infrastructure as a service (IaaS) and software as a service (SaaS). This service provides a wide range of 
facilities and benefits for consumers, among others, is the provision of self-service, elasticity, and pay 
per use [1]. Instant messaging is a technology that enables real-time text-based communication between two 
or more participants that utilizing the internet or intranet. A server that provides messaging services is 
commonly called Messenger [2]. 

Android based smartphone which introduced to the public in 2005 has became the most popular 
operating system with significantly increasing users each year. Based on survey report by Statista [3], in 
2013, over 967 million units of smartphones were sold to consumers worldwide and in the final quarter of 
2013, almost 78 % of smartphone sold is Android based smartphones. Based on unit shipments of these smart 
devices, Android’s market share increased significantly in 2014 with the company holding over 80 percent of 
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the global smartphone operating system market in the first quarter of 2014, and as shown on Figure 1, 
in 2017, 1.32 billion Android smartphones were sold around the world. 


2009 2010 2011* 2012 2013 2014 2015 2016 2017 


@ Android @ iOS RIM @ Symbian @ Microsoft @ Bada @ Other 


Figure 1. Statistics of android market share 


Social network, or Instant Messaging applications are being more widely used among users and new 
types of such applications are created by developers, such as WhatsApp, Viber, Facebook, Telegram, Line, 
WeChat, Beetalk [4], and Blackberry Messenger. Instant messaging became one of the popular smartphone 
feature with more than 1.4 billion users in 2015, and the growth in popularity of messaging apps is projected 
to continue. E Marketer, an independent survey agency, predicts that by 2018, the number of instant 
messaging application users worldwide will reach 2 billion and represent 80% of smartphone users, as shown 
on Figure 2 [5]. 


E Mobile phone messaging app users 
E% change 
% of mobile phone internet users 


Figure 2. Statistics of instant messaging application user 


One of the most popular instant messaging applications is Blackberry Messenger (BBM), although 
recently the use of BBM tends to decrease, but in some Asian countries, especially in Indonesia, BBM is still 
a leading application with lots of users, as shown in the results survey of GfK in Figure 3 [6]. 

In addition to the large amount of users, BBM also has numerous features, for example sending and 
receiving text messages, pictures, videos, and documents. BBM’s large amount of users and good features 
can be a magnet for someone who has criminal purpose such as drug trafficking, prostitution, cyber-bully, 
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and so on. There are some example of cases involving BBM applications in Indonesia as shown on 
Table 1 [7]. 
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Figure 3. Statistics of BBM users in Indonesia 


Table 1. Example of Digital Crime Cases using BBM 


No Year Case 
1 2014 Pornography using BBM at Banyuwangi 
2 2015 Parliament member’s BBM account hacked at Jakarta 
3 2016 Online fraud using BBM at Palopo 
4 2016 Identity theft using BBM at Palembang 
5 2017 Online prostitution’s transaction using BBM at Pekan Baru 


To solve digital crime cases involving smartphones, the investigator needs to do mobile forensics. 
Mobile forensics is science that performs the process of digital evidence recovery from a mobile device using 
the appropriate way with forensic conditions [8]. The investigator will conduct forensic analysis on the 
smartphone using some forensic tools with a forensically-tested methodology, the analysis results will 
become a supporting evidence that have validity value before the law and can be used as tool to solve digital 
criminal cases [9]. Primarily there are 3 different methods on mobile forensics acquisition techniques [10]: 

a. Manual Acquisition. In this technique the investigator will manually create the acquisition by directly 
looking at the contents of the smartphone device to find evidence. The investigator will takes pictures of 
each screen that containing the required data while browsing the device. The advantage of this technique 
is that it does not require any tools to conduct data acquisition, but this technicque also have 
disadvantages, the data that can be acquired is only the data that visible on the device and is time 
consuming. 

b. Physical Acquisition. In this technique the investigator will clone a smartphone device and conduct 
forensic analysis on the clone using a set of different forensic tools. 

c. Logical Acquisition. In this technique the investigator will conduct the data acquisition found in the 
smartphone device to be subsequently analyzed. Here data /information available on the phone is acquired 
using automated tools for synchronizing smartphone and PC. 

There are many challenges on mobile forensics fields, one of these challenges is the lack of 
resources, in the meaning that rapid development of mobile technology and the increasing amount of 
smartphone devices are not put in a balance by the development of forensic mobile technology and the 
existing forensic tools [11]. To overcome these challenges, a comparative analysis on instant messsaging 
features and forensic tools need to be done. The comparison is not only on forensic tool’s performance, but 
also on forensic frameworks such as National Institute of Justice (NIJ) [12], Hybrid Evidence Investigation 
[13], and Integrated Digital Forensic Investigation Framework (IDFIF) [14]. 

Sutikno, Handayani, and Stiawan et al [15] conducted a study to compare instant messaging 
features. The objects of the research is WhatsApp, Viber, and Telegram. The result of this research shows 
that WhatsApp is the most popular among the world’s users of the smartphones with about 60%, followed by 
Viber and, in third place, Telegram. Viber is the most functional messenger, but if the main concern is the 
security of communication, it is wiser to opt for Telegram. Telegram offers capability of syncronization, 
super fast service, reliable backup and better security feature. Although WhatsApp dominates the social 
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media space due to its simplicity and backed by giant i.e. Facebook, Telegram is essentially providing better 
platform than others.. 

On forensic field, Umar, Riadi, and Zamroni [16] uses Belkasoft Evidence Center, WhatsApp 
Key/DB Extractor, and Oxygen Forensic Suite 2014 performed comparisons and analysis of proprietary and 
open source forensic tools, the object for analysis is WhatsApp, a multiplatform instant messaging 
application, and the smartphone used for analysis is Android-based smartphone. The result of this research 
shows that Belkasoft Evidence Center has the highest index number, WhatsApp Key/DB Extractor has 
superiority in terms of costs, and Oxygen Forensic Suite 2014 has superiority in obtaining WhatsApp artifact. 

In other research conducted by Dogan and Akbal [17] Oxygen Forensic Suite 2014 and MOBILedit 
Forensics, The Researchers explain that every forensic tool has its own advantages and disadvantages. Digital 
crime cases related to smartphone devices should handled using several forensic tools that have different 
capabilities. The outcome of this research shows that MOBILedit Forensics has advantages in terms of run 
time, while Oxygen Forensic Suite 2014 has an advantage in terms of artifact analysis. 

Other comparative analysis research is conducted by Maurya, Awasthy, Singh, and Vaish [18] The 
Researchers using 2 proprietary forensic tools and 3 open source forensic tools, The Researchers conclude 
that many of the features that are present in proprietary forensic tools are also present in open source tools. 
Even there are certain features that provided by open source tool but proprietary tool does not, for example: 
SHA-1 hashing is not provided in EnCase but available in open source tools. Open source tools also have the 
advantages on cost, these tools are easy to buy due to no or negligible cost. 

Comparison and analysis of proprietary and open source forensic tools also conducted by 
Padmanabhan, Lobo, Ghelani, Sujan, and Shirole [19], the tools put into comparison are The Sleuth Kit 
(TSK) Autopsy, SANS SIFT, MOBILedit Forensics, and Cellebrite UFED. The conclusion of this research 
are: open source forensic tools have advantages in the number of users, flexibility in terms of use with 
console commands or GUI- based applications, logging capability, and good in tolerating errors. Meanwhile, 
proprietary forensic tools are superior in terms of process speed, data extraction accuracy, analytical skills, 
and ability to restore deleted data. 

According to research counducted by Salem, Popov and Kubi [20] using Cellebrite UFED and 
XRY, the outcome shows that XRY is better than Cellebrite UFED for acquiring most of the artifact types, 
while Cellebrite UFED is better on preserving the integrity of digital evidence. 


2. RESEARCH METHOD 

This research’s objective was to evaluate 3 forensic tools: Andriller, Oxygen Forensic Suite and 
Autopsy 4.1.1 based on framework and parameters from NIST and additional parameters from The 
Researchers in terms of the ability to acquire and analysis digital evidences from Blackberry Messenger on 
Android-based smartphone. 


2.1. Research tools and parameters 

The tools that used for this research are divided into two parts: Experimental tools and Forensic 
tools as shown on Table 2 and Table 3. 

The National Institute of Standard and Technology (NIST) has published a test plan to measure the 
performance of a forensic tool in a publication entitled “Mobile Device Tool Test Assertions and Test Plan 
ver. 2” [21] and “Mobile Device Tool Specification ver. 2” [22]. NIST provides a total of 42 measurement 
parameters and methods to measure the performance of forensic tools based on the results of each test plan. 
However, not all parameters were used in this research. Parameters that used in this research are shown on 
Table 4 and Table 5. 


Table 2. Experiment Tools 


No Experiment Tools Description 
1 Smartphone 1 Sony Xperia Z, Android Lollipop 5.1.1 
2 Smartphone 2 Samsung Galaxy A5 2015, Android Lollipop 5.0.1 
3 Blackberry Messenger Multiplatform Instant Messaging application 
4 Notebook Asus SonicMaster X450J, OS Windows 10 64bit 
5 USB Cable A data cable that can be used to connect smartphone to notebook 
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Table 3. Forensic Tools 


No Forensic Tools Description 


1 Andriller 


Windows-Based Proprietary Applications that can be used to 


acquire digital evidence on a smartphone 


2 Oxygen Forensic Suite 


Windows-Based Proprietary Applications that can be used to 


acquire digital evidence on a smartphone 


3 Autopsy 4.1.1 


Windows and Linux-based Open Source Applications that can be 


used to acquire digital evidence from multiple sources 


Table 4. NIST Core and Optional Assertion Parameters for Forensic Tools 


Mobile Device Tool-Core Assertion (MDT-CA) 


Core Assertion ID Test Assertion 


Comments 


MDT-CA-01 If a mobile device forensic tool provides support for connectivity of 
the target device then the tool shall successfully recognize the target 
device via all tool-supported interfaces (e.g., cable, Bluetooth, IrDA). 
If connectivity between the mobile device and mobile device forensic 
tool is disrupted then the tool shall notify the user that connectivity 


has been disrupted. 


MDT-CA-02 


MDT-CA-03 If a mobile device forensic tool completes acquisition of the target 
device without error then the tool shall have the ability to present 
acquired data objects in a useable format via either a preview-pane or 
generated report. 

If a mobile device forensic tool completes acquisition of the target 
device without error then subscriber and equipment related 
information shall be presented in a useable format. 

If a mobile device forensic tool completes acquisition of the target 
device without error then all supported data elements shall be 
presented in a useable format. 

If a mobile device forensic tool provides the user with an “Acquire 
All” device data objects acquisition option then the tool shall 
complete the acquisition of all data objects without error. 

Ifa mobile device forensic tool provides the user with an “Select 
All” individual device data objects then the tool shall complete the 
acquisition of all individually selected data objects without error. 
If a mobile device forensic tool provides the user with the ability to 
“Select Individual” device data objects for acquisition then the tool 
shall acquire each exclusive data object without error. 

If a mobile device forensic tool completes two consecutive logical 
acquisitions of the target device without error then the payload (data 
objects) on the mobile device shall remain consistent. 

Mobile Device Tool-Assertions Optional (MDT-AO) 
Test Assertion 

If a mobile device forensic tool provides the examiner with the 
remaining number of authentication attempts then the application 
should provide an accurate count of the remaining PIN attempts. 

If a mobile device forensic tool provides the examiner with the 
remaining number of PUK attempts then the application should 

provide an accurate count of the remaining PUK attempts. 

If the mobile device forensic tool supports a physical acquisition of 
the target device then the tool shall complete the acquisition without 
error. 

If the mobile device forensic tool supports proper display of non- 
ASCII characters then acquired data containing non-ASCII 
characters should be presented in their native format. 

If the mobile device forensic tool supports hashing for individual 
data objects then the tool shall present the user with a hash value for 
each supported data object. 

If the mobile device forensic tool supports acquisition of GPS data 
then the tool shall present the user with the longitude and latitude 
coordinates for all GPS-related data in a useable format. 


MDT-CA-04 


MDT-CA-05 


MDT-CA-06 


MDT-CA-07 


MDT-CA-08 


MDT-CA-09 


Optional Assertion ID 
MDT-AO-10 


MDT-AO-11 


MDT-AO-12 


MDT-AO-13 


MDT-AO-15 


MDT-AO-16 


Connect supported device via 
tool-supported interface(s); 
Acquire data. 

Begin acquisition; Disconnect 
interface or interrupt connectivity 
(.e., unplug cable) during 
acquisition. 

Acquire device data; Review data 
for readability in a useable 
format. 


Acquisition of MSISDN, IMSI, 
IMEI, MEID/ESN 


Acquisition of tool supported data 
elements 


Acquire all supported device data 
objects 


Acquire all supported device data 
objects by individually selecting 
each supported data object 
Acquire each supported device 
data object individually 


Perform two consecutive logical 
acquisitions; check mobile device 
for payload modifications 


Comments 
Input incorrect PIN; Check tool 
output for correct number of 
remaining PIN attempts Input 
Input incorrect PUK; Check tool 
output for correct number of 
remaining PUK attempts 
Physical Acquisition; Data is 
presented in a useable format. 


Acquisition of data containing 
non-ASCII characters 


Acquire data; Check known hash 
values for consistency 


Acquire data; Check GPS data for 
consistency 


Table 5. NIST Core and Optional Requirement Parameters for Forensic Tools 


Mobile Device Tool-Core Requirement (MDT-CR) 


Core Requirement ID Comments 


MDT-CR-01 A mobile device forensic tool shall have the ability to recognize supported devices via 
suggested interfaces (e.g., cable, Bluetooth) 
MDT-CR-02 A mobile device forensic tool shall have the ability to notify the user of connectivity errors 


between the device and application during data extraction 
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Mobile Device Tool-Core Requirement (MDT-CR) 
Core Requirement ID Comments 
MDT-CR-03 A mobile device forensic tool shall have the ability to perform a logical data extraction of 


supported data objects without modification 
Mobile Device Tool-Requirement Optional (MDT-RO) 


Requirement Optional ID Comments 
MDT-RO-01 A mobile device forensic tool shall have the ability to perform a physical data extraction 
for supported devices 
MDT-RO-02 A mobile device forensic tool shall have the ability to notify the user of connectivity errors 
between the device and application during a physical data extraction 
MDT-RO-03 A mobile device forensic tool shall have the ability to perform a physical data extraction 


(boot loader, JTAG, ISP) of readable memory without modification 


The measurement parameters are divided into 4 types, namely Core Assertions, Optional Assertions, 
Core Requirements, and Optional Requirements. Core Assertions leads to logical acquisition features and 
capabilities, Optional Assertions leads to physical acquisition features and capabilities, Core Requirements 
leads to logical acquisition requirements that a forensic tool shall have, and Optional Requirements leads to 
physical acquisition requirements that a forensic tool shall have. The Researchers does not include the 
parameters of MDT-CA-10 and the parameters on Universal Integrated Circuit Card (UICC) because the data 
on BBM application are stored in the internal memory, not UICC. 

There are several additional measurement parameters added by The Researchers as shown in 
Table 6. The additional parameters are more focused on the abilities of forensic tools to extract digital 
evidences from BBM for logical acquisition and physical acquisition that essential for forensic investigator 
during investigation of digital crime cases related to BBM. 


Table 6. Additional BBM Artifacts 
No BBM Artifacts 


1 BBM Account Profile 

2 Contact List (PIN Included) 
3 Conversation Data 

4 Images 


2.2. Research metodology 
This research uses the Mobile Forensic framework issued by the National Institute of Standards and 
Technology (NIST). NIST Mobile Forensic consists of 4 consecutive stages as illustrated in Figure 4 [23]. 


Figure 4. NIST mobile forensic stages 


Based on Figure 4, it can be described the mobile forensic analysis stages as follows [24]: 

a. Collection: This phase contained the process of identify, label, record, and retrieve data from relevant 
data sources by following data integrity preservation procedures. In this phase, no forensic tools used 
since forensic examiners will conduct the investigation based on physcal data on physical evidences. 

b. Examination: In this phase actual data is gathered from physical evidence. In an ideal case the data is 
forensically copied from the phone as well as from the SIM Card. In some cases technical diffculties can 
prevent a digital accusation of the device. In a worst case scenario only screen captures of the phone can 
be gathered. 

c. Analysis: Analyze the results of the examination by using technically and legally justified methods to 
obtain useful information and answer the questions that encourage the collection and examination. The 
analysis conducet is not only how to present the digital evidence as a tool on court, but also how to 
determine forensic tool’s performance that used on physical evidence. 
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d. Reporting: The last step is the most important. This phase is the presentation of the outcome of the whole 
process in a conclusive manner and offer the other party information about the forensic tools evaluation 
and methods used. 


3. RESULTS AND ANALYSIS 

The functionality of NIST Mobile Forensics Framework is not limited to extracting and retrieving 
digital evidence as a tool to resolve digital crime cases that presented in court, but this framework is also can 
be implemented on a comparison analysis of forensic tool performance as conducted in this research. The 
results of the comparison analysis of this forensic tool will be presented at the reporting stage. The stages of 
comparative analysis conducted using NIST Mobile Forensics Framework are described as follows: 


3.1. Collection 

At this stage, collection and data recording of physical evidence is conducted. This physical 
evidence data collection process includes the image of physical evidence, brands, specifications, operating 
systems, IMEI, and other data that can be extracted from physical evidences without using any forensic tools. 
In this research the collected physical evidence is in the form of 2 android-based smartphones. The result of 
this stage is as shown on Table 7. 


Table 7. Physical Evidence’s Specification 
Physical Evidence 1 


Brand Sony 

Serial Xperia 

Model Z 

Model # C6602 

IMEI 355666050620xxx 
OS Android 


Version 5.1.1 (Lollypop) 
Processor Quad core 1.5 GHz Krait 


Physical Evidence 2 
Brand Samsung 
Serial Galaxy 
Model A 
Model # SM-A500F 
IMEI - 
OS Android 


Version 6.0.1 (Marshmallow) 
Processor Quad core 1.2 GHz Cortex-A53 


3.2. Examination 

Examination is the process of physical evidence backup and retrieval of digital data that contained 
in it. At this stage the cloning process of physical evidence is conducted. In this research, the cloning process 
conducted by using MOBILedit Forensic Express [25]. MOBILedit Forensic Express is a tool with backup 
and cloning features, by using this feature, forensic examiners are able to maintain the integrity of physical 
and digital evidences. The process and the result is as shown on Figure 5. 


|_| LENOVO Lenovo A390_ROW (2018-02-02 22h17m25s) 
B Physical image - Sony C6602 (2017-08-12 10h51m03s) 
a Samsung Galaxy A5 (2018-02-12 04h11m28s) 

|_| Sony Xperia SL (2018-02-03 18h48m37s) 

EE LENOVO Lenovo A390_ROW 

WB Samsung Galaxy AS 


E Sony Xperia SL 


Figure 5. Cloning process and results 
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In this stage, the retrieval process of digital evidence will be conducted by using Andriller, Oxygen 
Forensic Suite, and Autopsy 4.1.1. 


3.2.1. Andriller 

Examination process that conducted using Andriller resulted an integrated HTML report that 
contained all the data extracted from physical evidence, the examination process for both physical evidences 
shown on Figure 6. 


Global Output Location (Decoders / Extraction / Parsing) Global Output Location (Decoders / Extraction / Parsing) 
Output E\AnanlienXpena Z - Andriier - Jul 09 2017 Output EsAndrilier'Samsung A5 2015 
Extraction (ADB) Data Parse (Folder) Date Parse (TAR) Date Parse (A8) Extraction (ADB) Data Parse (Folder) Data Parse (TAR) Data Parse (A8) 


Cect Serial ID: Bx903FBCZH Check Serial ID: 7bcad0a2 


nerel Device Information °° 
H 


Bulld nunber 

Wi-fi MAC: @O:eb:20:33:ba:2a 

kocal tine: 20: 5 00:00:00 SE Asia Standarc Tine 
19 05:17:19 WIB 


20 
00:00:00 SE Asia Standard Time 
2 03:01:17 WIS 
Synchronised Accounts °° 
M Groups 

~donsyahSgnail .com 

oo jata Extraction via Android Backup ° 
>>> Extraction via Android Backup method <<< 

>>> Unlock the screen and tap on “Back up my data” <<< 


een xtraction via Androld Backup *e#*teteesensee 
>>> Extraction via Android Backup method <<< 
>>> Unlock the screen anc tap on “Back up my data" <<< 

Clear Leg Seve log 


Reading backup: 123.2KB read Days lett 3 


Clear Log Savetog 


Reading backup: 9.1MB read Days it 11 


Figure 6. Examination process using Andriller 


Forensic examiners then will be able to navigate through the generated HTML report to find digital 
evidence needed. The result from both physical evidences shown on Figure 7. 


[Andriller Report] SONY C6602 | IMEI:355666050620508 [Andriller Report] SAMSUNG SM-A500F | IMEI:Unknown 


ADB serial: BX903FBCZH ADB serial: 7bcad0a2 
Shell permissions: —_ |root(su) Shell permissions: [shell 
— ponr Manufacturer. [SAMSUNG 
Model: 

oo co Moder: SM-A500F 
IMEI: 355666050620508 

a z IMEI: Unknown 
Android version: 5.1.1 
= Android version: |5.0.2 
Build name: - 
Wi MAC 00:eb:2d:33:bd:2a Build name: 
Localtime: 2017-07-15 00:00:00 SE Asia Standard Time Wifi MAC: Tcf9.0e:1d:07:20 
Android time: 2017-07-09 05:17:19 WIB Local time: 2018-02-05 00:00:00 SE Asia Standard Time 
com.sonyericsson.localcontacts: Phone contacts Android time: 2018-02-12 03:01:17 WIB 
s com.google: arizona.it2013@gmail.com 3 
—— com.bbm.account: BBM Groups Accounts: home sees apr ier Gaps: 
com.whatsapp: WhatsApp com.google: afirdonsyah@gmail.com 

Filesystem: Shared Storage (2,221) Filesystem: Shared Storage (29 
‘System: Wi-Fi Passwords (2) System: Wi-Fi Passwords (0 
(Communications data: |SMS Messages (2) System: Android Download History (6) 
Applications data: Blackberry Messenger (158) ‘Applications data: | Blackberry Messenger (36) 


Figure 7. Andriller’s examination result 


The examination result then will be analyzed and compared to other forensic tools used in this 
research and the comparative analysis result will be presented in Reporting phase. 


3.2.2. Oxygen forensic suite 

Oxygen Forensic has the ability to perform logical acquisition and physical acquisition. 
Examination process that conducted on both physical evidences using Oxygen Forensic Suite as the forensic 
tool is as shown on Figure 8. 

Examination result that acquired using Oxygen Forensic Suite provide complete data of physical 
evidence that contained Device Information, Forensic Examiner’s Identity, List of Contact, and Installed 
Application, BBM included. Figure 9 showed examination result from both physical evidences. 
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Same with Andriller, Forensic Examiners then will be able to navigate through this generated report 
to find digital evidence needed. This examination result also will be analyzed and compared to other forensic 
tools used in this research and the comparative analysis result will be presented in Reporting phase. 


Processing physical dump 


Physical arp s beng created 
Extracting data from E: (4OER edit Samsung Galaxy ASG 


RETTET] 
Extracting RELOS Paian eye ih CN 
data. Extracting 


data. 


Warning! The data is being extracted from the device right now. 
Do not disconnect it or make any changes to the device. 


whee cance w Heo 


Figure 8. Examination process using oxygen forensic suite 
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e TR te aromer AL eo tre restos (fee J ay Batten a Bons 1B cao sere enon eg € Syhiien 
a — rper Qeni Jani 
HD ura ene siets wept — serte S auni Be mmp a... Qi i 
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Figure 9. Oxygen forensic Suite’s examination result 


3.2.3. Autopsy 4.1.1 
Autopsy does not have data examination feature for the Android platform, digital evidence 


examination process can be done through image/cloning of physical evidence (logical examination). The 
examination result of both physical evidence is as in Figure 10. 


Raport À — ©-remiiss t Directory Listing 
: Images 
(a Tabie Thumbnail 
rages 
Page: 1of 16 Pages: >) Goto Page: Images: 1-200 ‘Small Thumbnails v 


com.sec.android... 


Figure 10. Autopsy 4.1.1’s examination result 


3.3. Analysis 
Analysis is a stage to check and compare Examination result thoroughly to get the performance 


analysis forensic tool used. This stage limits the searching process to a certain point that connected to certain 
data or application. At this research, the search limit is BBM. 
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Based on the examination process conducted, Andriller was able to conduct physical acquisition 
only. Andriller have many shortcomings in terms of Core Assertions and Optional Assertions. The 
examination result shows that Andriller was able to get some information regarding smartphone devices, 
such as IMEI (International Mobile Equipment Identity), ADB Serial, Manufacturer, and Android version. 
From the NIST parameters used, Andriller succeeded in meeting the criteria of MDT-CA-01, MDT-CA-02, 
MDT-CA-03, MDT-CA-04, MDT-CA-05, MDT-CA-06, MDT-AO-12, MDT-RO-01, MDT-RO-02, and 
MDT-RO-03. As for additional parameter added by The Researchers, Andriller was able to acquire digital 
evidence in the form of Conversation Data as shown on Figure 11. 


[Blackberry Messenger] [Blackberry Messenger] 


jaaezas4 |45160730 


Figure 11. Andriller’s conversation data 


As for Oxygen Forensic Suite, this forensic tool was able to conduct both physical and logical 
acquisition. The examination result also provides information on smartphone devices, such as IMEI, 
Manufacturer, and Android version. From the NIST parameters used, Oxygen Forensic Suite suceeded in 
meeting the criteria of MDT-CA-01, MDT-CA-02, MDT-CA-03, MDT-CA-04, MDT-CA-05, MDT-CA-06, 
MDT-CA-09, MDT-AO-12, MDT-RO-01, MDT-RO-02, MDT-RO-03, and MDT-CR-03. As for additional 
parameters added by The Researchers, Oxygen Forensic Suite was able to acquire all type of additional 
parameters from both physical evidences as shown on Figure 12, Figure 13, Figure 14, and Figure 15, as in 
accordance to MDT-AO-13 NIST Parameter. 
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Figure 13. Oxygen forensic suite’s BBM chat artifact 
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Figure 15. Oxygen forensic suite’s BBM image artifact 


The analysis performed on Autopsy 4.1.1’s examination result shows that Autopsy was able to do 
logical acquisition only, as in accordance to MDT-CA-09 and MDT-CR-03 NIST parameter. Analysis 
conducted in term of additional parameters added by The Researchers does not give the expected result 
because Autopsy 4.1.1 does not have the file decryption feature to open the encryption of BBM database file, 
in other words, extraction using Autopsy gives zero result. 


3.4. Reporting 

The last stage on NIST mobile forensic framework is reporting. At this stage all the analysis’s result 
will be presented in detail and all analysis result related to forensic tool performance comparison that 
obtained from BBM application is documented. The report will be presented in the form of comparative table 
based on NIST Parameters as shown on Table 8. 


Table 8. Evaluation Results 
Measurement Parameters Forensic Tools 
Andriller Oxygen Forensic Suite Autopsy 4.1.1 


Core Assertions MDT-CA-01 
MDT-CA-02 

MDT-CA-03 

MDT-CA-04 

MDT-CA-05 

MDT-CA-06 

MDT-CA-07 - - - 

MDT-CA-08 - - 

MDT-CA-09 - V V 

Optional MDT-AO-10 - - - 
Assertions MDT-AO-11 - 
MDT-AO-12 V 

MDT-AO-13 - 

MDT-AO-15 - - - 
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Measurement Parameters Forensic Tools 

Andriller Oxygen Forensic Suite Autopsy 4.1.1 

MDT-AO-16 - - - 

Core Features MDT-CR-01 - - - 
Requirements MDT-CR-02 - 
MDT-CR-03 
Optional Features MDT-RO-01 
Requirements MDT-RO-02 
MDT-RO-03 


aes! 
2eee' 
1 


Andriller is only capable of conducting physical acquisition. However, Andriller successfully 
obtained BBM Conversation Data. From experimental results using Oxygen Forensic Suite, almost all core 
parameters and optional of NIST are met entirely. Autopsy 4.1.1 did not meet all the NIST parameters except 
for parameters related to logical acquisition. 

On additional parameters added by The Researchers, The Researchers used calculations with index 
numbers to determine the performance of each forensic tool in accordance with the experiment results. The 
calculation of index number used is unweighted index as shown in Equation (1). 


Zaro 


Par = YLarT 


x 100% (1) 


Table 9 shows the results of performance analysis conducted on each forensic tool. Andriller got 
25% performance index score by only managed to acquire 1 type of BBM artifact, Oxygen Forensic Suite got 
100% performance index score because it successfully acquired all 4 types of BBM artifacts, and Autopsy 
4.1.1 did not get any artifact (zero result). 


Table 9. Performance Index Scores 


Physical Evidence 1 
Forensic Tools 


NO BEM Artilact Andriler Oxygen Forensic Suite Autopsy 4.1.1 
1 Account Profile - y - 
2 Contact List - V - 
3 Chat V V - 
4 Image - V - 
Performance Index Score 25 % 100 % 0% 
Physical Evidence 2 
; Forensic Tools 
No PEM ete! Andriler Oxygen Forensic Suite Autopsy 4.1.1 
1 Account Profile - Vy - 
2 Contact List - V - 
3 Chat V V - 
4 Image - V - 
Performance Index Score 25 % 100 % 0% 


Related to the evaluation results based on NIST parameter criterias, The Researchers used the same 
formula to determine the performance of each forensic tool in accordance with the experiment results. Based 
on the calculation conducted, Andriller has performance index value of 47.61%. Oxygen Forensic Suite has 
performance index value of 61.90%. Autopsy 4.1.1 has performance index value of 9.52%. 


4. CONCLUSION 

Based on parameters added by The Researchers, Oxygen Forensic Suite has the highest index 
performance score at 100%, followed by Andriller with index performance score at 25%, and Autopsy 4.1.1 
did not give any result (zero result) due to the absence of file and image decryption feature for mobile device. 
Related to NIST parameter criterias, Oxygen Forensic Suite still has the highest index performance score at 
61.90% and meets almost all NIST parameter criterias. Andriller is on the 2nd with index performance score 
at 47.61% and meets 10 NIST parameter criterias. Autopsy 4.1.1 has the lowest index performance value at 
9.52% due to the absence of document decryption feature and only meets 2 NIST parameter criterias. 
Andriller indeed met many NIST parameter criterias, however, Andriller manages to get only conversation 
data artifact using physical acquisition. Oxygen Forensic Suite has the highest performance score among the 
three forensic tools used, but Oxygen Forensic Suite has weakness in terms of options to select data for 
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acquisition due to limited options on data extraction menu. Oxygen Forensic Suite successfully extracts all 
BBM artifact via logical acquisition and physical acquisition. For future work, there are more performance 
evaluations on forensic tools that can be conducted to get an overview on what forensic tool that best for 
digital forensic investigations. 
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